845 research outputs found
Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing
The question of what kinds of linguistic information are encoded in different
layers of Transformer-based language models is of considerable interest for the
NLP community. Existing work, however, has overwhelmingly focused on word-level
representations and encoder-only language models with the masked-token training
objective. In this paper, we present experiments with semantic structural
probing, a method for studying sentence-level representations via finding a
subspace of the embedding space that provides suitable task-specific pairwise
distances between data-points. We apply our method to language models from
different families (encoder-only, decoder-only, encoder-decoder) and of
different sizes in the context of two tasks, semantic textual similarity and
natural-language inference. We find that model families differ substantially in
their performance and layer dynamics, but that the results are largely
model-size invariant.Comment: Accepted to BlackboxNLP 202
Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers
Scaling analysis is a technique in computational political science that
assigns a political actor (e.g. politician or party) a score on a predefined
scale based on a (typically long) body of text (e.g. a parliamentary speech or
an election manifesto). For example, political scientists have often used the
left--right scale to systematically analyse political landscapes of different
countries. NLP methods for automatic scaling analysis can find broad
application provided they (i) are able to deal with long texts and (ii) work
robustly across domains and languages. In this work, we implement and compare
two approaches to automatic scaling analysis of political-party manifestos:
label aggregation, a pipeline strategy relying on annotations of individual
statements from the manifestos, and long-input-Transformer-based models, which
compute scaling values directly from raw text. We carry out the analysis of the
Comparative Manifestos Project dataset across 41 countries and 27 languages and
find that the task can be efficiently solved by state-of-the-art models, with
label aggregation producing the best results.Comment: Accepted to EMNLP 202
Vetluguin as Frederick Van Ryn: about Vladimir Ryndzyun’s American Literary “Projection”
The article examines the American period of V.I. Ryndzyun's life and his work as an American writer and journalist. Vladimir Ryndzyun began publishing in Russia before the Revolution, and in the early 1920s he became famous under the pen-name “A. Vetlugin” thanks to publications in Russian émigré newspapers and books published in Paris and Berlin. He arrived in the USA in 1923 with Sergey Yesenin and Isadora Duncan and decided to stay there. He contributed to American journalism and American cinema as Voldemar Vetluguin, and to American literature as Frederick Van Ryn. In the fall of 1933, his essay “There's No Repealing Tastes”, signed with the pen-name Frederick Van Ryn was published in the first issue of Esquire magazine along with the works of Ernest Hemingway, John Dos Passos, Erskine Caldwell, etc. In 1934 he became an editor of RedBook Magazine as Voldemar Vetluguin. In 1936 he started to contribute in this pulp fiction magazine as Frederick Van Ryn as if he decided to separate his work as an editor and his literary works. Under the name of Frederick Van Ryn, he also collaborated with other magazines: a number of his publications appeared in the 1940s in Liberty; all of them have to do with cinema. His new pen-name made it possible for him to develop the author’s strategy in the USA, but in general it fits into the system of author’s masks / projections, which are typical for V.I. Rynzdyun’s / Voldemar Vetluguin’s works and self-fashioning
- …